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ABSTRACT 

Evaluation affects the people who participate and the 
situation studied, and there is no guarantee that the effects will be 
democratic, constructive, or equitable. A case study illustrates that 
even the most scrupulous attention to ethical principles aiid 
procedures for the release of information cannot always protect the 
interests of participants in the study, or the integrity of the study 
itself. At issue was a male school principal instructing a female 
teacher to change the record of what she had said in evaluating a 
language curriculum project. The objectives of the paper are the 
following: (1) to demonstrate the way in which local reactions to an 
evaluation directly threatened its validity; (2) -to demonstrate that 
not all participants in a. qualitative evaluation study understand or 
defer to the commitments of democratic evaluation; (3) to indicate 
that the metapolitical analyses of evaluation approaches do little to 
inform the nature of local site effects; and (4) to identify some new 
theses about the practice of "democratic" evaluation. (MLF) 
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Introduction 

Evaluation is a reflexive activity. It affects the people who participate and tiie situation studied, 
and theie is no guarantee that the effects wiU be democratic, constra^ Most of 

the critique of evaluation involves analysis of the political relationships between program 
participants, program sponsors, progr^ evaluators and their respective institutional 
affiliations. Arguments for 'qualitative', 'naturalistic' and 'case study' approaches to^ 
evaluation are based oh the importance of all mvolvcd and afifectwi participants' interpretations 
of life m social and education^ programs* Such approaches are justified on two main grounds: 
political and substantive* Political grounds attribute participants 'equal rights' to comment on 
programs* Substantive grounds recognise the validity of inteiprstations of the program is not a 
function of the relative pow^r and status of researchers, policy-makers, and program workers* 
Rather the opposite is that case, the validity of the evaluation is regarded as suspect unless 
different perspectives, descriptive and evaluative, are given expression and taken into account 
But these analyses do not engage the ways in which any 'external' evaluation itself impinges on 
the lives of program participants at the 'local' level* 

The first objective of tiiis paper is to demonstrate the way in which local reactions to an 
evaluation directiy threatenea its validity. A second objective is to demonstrate that not all 
participants in a qualitative evaluation study (which airned to be democratic, but within certain 
institutional and procedural constraints) understand or defer to die commitments of democratic 
evaluation* A third objective is to mdicate tiiat the metapolitical analyses of evaluation 
approaches do littie to inform die nature of local site effects. MacDonald's (1976) well-known 
distinction between 'democratic', 'bureaucratic' and 'autocratic' epproaches invites a view of 
democratic approach of evaluation which results in so-called democratic evaluation approaches 
which may not be democratic at all. There is an inoiportant distinction between 'participatory' 
and 'representative' democracy which is too often disregarded. A fourth objective is to identify 
some new theses about die practice of 'democratic' evaluation, to improve the way 'evaluation' 
understands the nature of social and educational change, to improve the relevance of evaluation 
to social and educational improvement, but at die same time to suggest tiiat what is 'democratic' 
is very situational and made problematic by concrete practice. 

\ The case 

X The following account of an instance within a democratic evaluation which used a case study 

^ approach illustrates diat even die most scrupulous attention to etiiical principles and procedures 

^ for the .release of information cannot always protect the interests of participants in the study, or 

the integrity of the study itself Departures from die principles and procedures by participants 
X left the evaluation team widi intensely problematic alternative courses of action. The account 

\ has been anonynused and certain unimportant features have been fictionalized for reasons which 

will become obvious. 
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The evaluation study was one of several I have participated in over recent years. The Principles 
of Procedure for the control of the release of information by the evaluation were an early 
version of those developed and subsequentiy published by Kemmis and Robottom (1986). 
Those authors note that the principles have been observed for several years in a variety of ways 
by evaluators in the naturalistic case study evaluation tradition (see also Sunons (1987)). The 
principles used are thought to provide a way of making an externally commissioned eviuation 
as democratic as possible by giving participants considerable control over the interpretation and 
release of information. With respect to the issues addressed here, the Principles of Proceduie 
used m the study were not significandy different from the published Kemmis and Robottom 
(1986) version (see Appendix A for the actual principles used). 

For the purposes of tiiis account, it is sufficient to describe the program being evaluated as the 
•Language Curriculum Project', where students with particidar weaknesses m language were to 
visit a neighbouring school for specially funded language tuition. The project was one of many 
in the past decade or so supported by the Australian national and state governments which 
aimed to offset educational disadvantage of different kinds with specially funded programs. 

Because the evaluation study mvolved oiJy a few sites, each of which was quite distinctive, the 
anonyraisation of participants in reports of the study itself was not considered feasible. This 
was so even though the sites were quite widely distributed around Australia. While it might 
have be.possible to disguise thv? identities of individuals to people without knowl^ge of 
particular sites, people at each site would almost certauily have been recognizable to site 
colleagues because of the distinctiveness of their roles and other cues. Because it was 
effectively unavoidable, in my communication with participants in the evaluation I expressed a 
preference for identifying participants in die evaluation report, but only with their permission. 
I asked people to release inforaiation with its eventual public character m mind (see letters 
below). 

The wide distribution of sites also made the 'case' itself rather attenuated. This was 
compounded by die intermittent nature of the program's activities, and by the substantial 
departure of program activities from the intended goals of the program. The program was so 
much reuiterpreted by site participants it was not always clear whether what was happening was 
an example of the 'case' we were commissioned to study. We agonised several times as to 
whether we should report that observation, tenninate the study, and return the grant But the 
way the program had panned out seemed to us to be a useful tiling to understand, so we . 
proceeded. 

In a superficial sense, die 'boundaries' of die case were reasonably clear. We simply had to 
find where die program money had gone to. That was not difficult, but finding out how its 
purposes had been so obscured or distorted before it actually got to diose who were meant to 
make use ofit was anotiier kettle offish. Furthermore, die sheer geographical extent of die 
case strained the budget for die evaluation. This made fieldwork radier compacted, and all of 
diese factors combined influenced die possibilities and nature of die democratic practice die 
evaluation team could engage. We ttied, but I have grave reservations about how successful we 
were. 



The instaiice 

The two key actors m die account are die male school principal, and ±e female language teacher 
employed to work on die I^^^uage Project in an untenured position - on 'soft money'. Bodi 
were made aware of die Pri !es of Procedure in general terms, but were not given actual 
copies of tiiem in advance, vy iien interviewed the principal was amiable, helpful, apparendy 
supportive of die Language CHirriculum Project, appreciative of die 'responsive' approach 
taken by die evaluators (Stake, 1975), and indicated his preparedness to help die evaluation in 
any way he could. He spoke of die Language Project in glowing terms, and was proud of die 
school's engagement in it. He indicated tiiat he was not aware of any of die operational details 
of die Project and encouraged me to speak widi teacher directly involved. It was clear also Irom 
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what he said that other teachers were unlikely to have any knowledge of die project From his 
perspective the project was a good idea, but the details were left to two staff, one of whom was 
not available to die evaluation team. 

The teacher^s account of the Project was not so positive* She criticized the Project severely on 
three grounds which can be summarized in the following way* First, the curriculum which she 
had b^n encouraged to develop had not really been matched to the kinds of disabilities the 
tfitended students had; though she had never been told exactly who the mtended students were 
meant to be* Second, the students were being selected in ways which meant they were not die 
cliait group for whom funds had been provided And diird, die organization of die Project was 
a 'disaster,* with students were arriving unannounced, at the wrong times, from schools which 
hadhot uidicated participation m die Project and which would not have been die piimary target 
schools for die Project The verbal information she presented to support her claims was 
supported by enrolment records, by our observation of the students, and by the students* own 
comments. 

The activities die teacher constructed for die students who did arrive seemed engaging and 
useM ID die students, emd her mgenuity and calm in coping widi die surprise arriyal in die 
school foyer of a troupe of students firom odier schools was remaikable* Neverdieless, after the 
event, she expressed annoyance at the inconvenience she had been caused, and also at what she 
saw as the misdirection of project fuiids (several other teachers we interviewed at other sites had 
been largely miaware of any of die commitments accepting die funds entailed because diey did 
not where the fimds actually came from)* This teachei^s criticism criticism was so strong, I 
asked her at the tinoe whether she was sure diat her comments should go on the public record 
and be attributed to her She was adamant about that, saying that it was important for people to 
realize what had happened I still had reservations, but almost felt obliged to use what she had 
said. 

The Principles of Procedure which guided die study dictated two phases for die negotiation of 
the release of information. The first phase involved the negotiation of the substance of 
interview and other observational data. The second phase invited particij^ants to react to the 
interpretations made by the evaluator in the construction of the draft version of die final report 

The teacher and principal were each sent copies of die record of the interview conducted, a copy 
of die Principles of Procedure , and the following letter 

Dear , 

Please find enclosed my edited record of our discussion during my recent visit. 
As I may have indicated when we talked, the evaluation team like to give 
participants in the study the opportunity to correct or improve our record of their 
statements. Please note in the enclosed outline of the study the 'Principles of 
Procedure * to which we are committed. 

We are concerned that the record is fair, relevant and accurate. We would also 
like participants to indicate any comments which are an accurate record, but 
which they would prefer not attributed to them personally. Our strategy with 
such comments if they are reported in the study will be to, disguise the identity of 
the utterer - 'a teacher reported ... 'or 'a Senior Officer in the Education 
Department in one state said ... It is our preference to identify people, but we 
will not do so without consent. We will not identify any students in reports of 
the study. 

We would be grateful for any corrections which need to be made, and for any 
further information which you believe will assist our understanding. If we do 

not receive an amended statement from you brfore , we will assume 

that you are coprfortable with the document as it now stands. Would you please 
address your response to me as follows: 
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Dr Robin McTaggart 
School of Education 
Deakin University 
Geelong 

Victoria, 3217 (Telephone: 052 471448 or 052 471483. for messages). 
Thank you for your help. 
Yours faithfully, 
Robin McTaggart. 

SrKS^ rlr?^ of interview with minor, perhaps pedantic corrections. The 

r^^^^^Z^-^ ^^l in retimung her reply, so I contacted her by telephone. She suggested a 
^^^g^s/o'^ clanfication and a few grammatical dbrrections, for example, 
"^Z^ T ^ • reaffirmed that it was important her comSients 
were on the record. I asked her to send a copy of her corrections to confirm what had been 
!Si5'»? f? °"®L?^® agreed that the final report writing could go ahead, and seemed 
surpnsed that I was being so cautious. She saw no problem at aU with my using the 
intoiaoon and expUddy attested to the fairness, relevance and accuracy of our iecord of her 
wOTds.. Indeed, she said again that she had given it to us for the very purpose of pubUcation 
SlS'^i^^H-^^^^A"^*"?^^^^^ Wedidnothaveasitesmtementofher ' 
^nf tl^l^^^T ^""^ ^« by the Principles of Procedure 

A. statement could be used We decided to quote her to encapsulate 

the lands of things people at several sites we had visited were saying? 

At tills time the writing of the draft final report was weU under way. It was in the maU to studv 

SiSw h Jn^S ^^^-w!?' P^^"??^^ confirmation of tiieW^s^^S ^ 
mterview had not been received before mailmg, but I proceeded on tiie basis of her assurances 
foSg Semo:'°"'''^^^^^ Diaf^ final reports were sent to stiidy participants witii ST 



Language Curriculum Project Evaluation Study 
Draft Report 

The enclosed draft of the final report ofth^ evaluation of the Language 
*-urnculum Project has been circulated only to those participants who have been 
quoted directly. In all but one or two cases, the comments attributed to study 
participants have been cleared with them already. All participants identified in 
the study are invited to check their own comments arJ interpretations made by 
the Language Curriculum Projea evaluation team. 

The Principles of Procedure to which the Language Curriculum Project 
Study IS committed are included in the report itself as an Appendix 
Participants may wish to refer again to those principles before deciding whether 
to respond to this drf report. The authors' basic commitment is that the report 
be fair, relevant and accurate, and invite participants to read the report with 
these cntcTid m jtiittd^ 

The report will be amended according to the Principles of Procedure in the 
light of comments received, and will then be published. 

Written comments on the drefi report should be directed to: 
Dr Robin McTaggart 
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School of Education 
Deakin University 
Geelong 
Victoria, 3217. 

The deadline for responses is Beyond that date the authors 

will assume that participants who have not responded approve the release of the 
document in its present form. 

Robin McTaggart may be contacted by telephone at Deakin on 052 471448 or 
052 471483 for messages. 

Thank you once again for your contribution to the study. 

A few days after the draft report arrived at the various sites, an amended record of intendew 
was received from the teacher. The changes she had made were extensive, with all of the 
criticisms of the Language Curriculum Inject which had been previously released either 
deleted, or so completely submerged that they were virtually imperceptible to the reader. 
Important testhiK)ny for the evaluation was now apparently withdrawn. I was concerned about 
the loss of data, but because of the teacher's earlier attitude I was even tcxm concemed about 
what might have happened I was also curious about the turn of events. The site was too 
distant to visit again, so I telephoned her to find out 

She was angry, but not with me* When tiie principal had read tiie draft fined report, he had also 
read the interview which tiie teacher had previously approved for release (by teleph<)ne). He 
had summoned her to his office and instructed her to change tiie record of what she had said 
The reasons she reported to us were that tiie principal tiiought tiie interview reflected adversely 
on tiie school and implicitly on his supervision of tiie staff member re'spoiisible for tiie 
organizational arrangements* That staff member apparentiy had been s^sent iiom tiie school 
periodically witiiout permission from tiie principal. However^ according to tiie Project teacher, 
the fairness, relevance and accuracy of her testimony were not questioned by tiie principal, at 
least in tiiese terms. He wanted to change her words even so. 

The principal's actions were in direct violation of ♦he Principles of Procedure to which he had 
agreed in general at our first meeting, and which he now had in &ont of him having seen tiiem 
also when his interview was returned to him for ;x>mment My first impulse was to contact him 
but I decided against it for several reasons. I had interviewed him and my records of the 
mterview witii him wd:e negotiated according to tiie Principles of Procedure. He said he did 
not know much about tiie operation of the program which is probably not unusual in such 
cases. At tiie time he had tiie reported conversation witii tiie teacher about tiie content of tiie 
interim report, he had before him a writtai invitation to contact me to suggwt .changes to tiie 
interim report's data and interpretations witii reference to tiie criteria of fairness, relevance and 
accuracy and tiie Principles of Procedure in tiie letter. He did not respond in that way but 
instead apparentiy decided on tiie course of action reported - to force tiie teacher to change her 
testimony. 

We tiiought his reaction was excessive, but he may have seen himself in an invidious position. 
What ri^ts did he have now tiiat tiie evaluation had actually increased his understanding of 
what had been happening in tiie program? Tliat is, the Principles of Procedure might have 
underestimated tiie dynamism of tiie site, botii substantively and politically. If tiie principal 
acted to change tilings immediately, how could tiie teacher's account any longer be true? But its 
earlier trutii was important to tiie evaluation. 

Referring to her rights under tiie Principles of Procedure, I asked tiie teacher if she had resisted 
the principal's efforts to force her to change her comments. She relived the anger of the 
confrontation and assured me: 'I know how to dig my heels in!' She went on angrily: 
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Do you know what? He even mentioned my tenns and conditions of 
employment! Can you beUeve that? I wouldn't mind so much except that if I 

htvl choice ""^^^ ^ ^° S° ^^^'1^0"? 

I considered sticking to the original release agreement, but felt it unfair to do wh«»n th^ n„t« 
potential victim was reluctant to do so. I suggested we St mr^^^ om^fZ, of 
which was accratable to her, and which might stiU makeTe S sh^^^^^ 
SS?^ She was happy about that, but indicated her dSo SS^^iSte ^'th Se 
CS?.?.^''- • °«J«g to her. this made matters worse. His reaction wasTthrow his 
^^.^I'^u^i-VJ'^'' 'Write what you damn well like!' Knowing^ iSations 
the Project teacher decided to go back to a form of words veiy clo,«« to that whichsSd ri^e 
principal had imposed upon her in their first confi^ontation. 

h^^SMd^^lS^c'H^^^^ "^^^"1 ^° ^^^"ation is an important 

So ■ u » i • . "2"^ to know' take precedence over the individuals' riehts to 'own the 

fssueS ^"'^ ' ^ P"^ Of course S^o^iX^^^^ 

Aru,^h*.'rS±'ll^ '^f ^ '^J^^ ?^P^^ to us to indicate in the evaluation report 
?n?^nl- circumstances (without the influence of an evaluation team) theproSwas 

not worfang weU as a central participant saw it. Periiaps the Principles oflSi? nSS? 
tt>D much away miyway. I felt committed not to publisS the origin^ S^^^^ 
^^^7 "f^^ ''^^^y i°»Plicit claim of the teachS^Sd teSaU 

Issue 1: The 'truth' of the teacher's testimony 

We were satisfied that tiie teacher's original testimony was accurate There was conoboratinir 
f^^l^'^^^n was not uncommon in the sites we S visited SSc^f^^ 
^etong students firom school to school, identifying appropriate participms and eSg their 
Z&^^'i '"r' demonstrably beyond the resoloces of the sahboS^^^^^^ 
^ence dfd^^K^^'i concerns AepSc^S^£d)^SCt 

m only other informant who could help out there was the principal who had opportunity to 
present his side of the story': he was interviewed, and coiroborated our rSor^K^a 
rS^^™ I" f^"' ""{^ ^° to us by telephohe or in ZS^g^±Tv^^e of 

Ae altercation with the teacher. His statement indicated that he did notSolSevdStSl 
about the Fogram. so he could not refute what she said The evSenre we h^ve dS^^ 
exptoins how we dealt with the situation and how we in^^enTr^tS it T^^^^^ 

nouZZ &£?h*' ^^i' '"^ -^^.^^^ afflr<^nvS^ could 

teach^s^ai^S ^ "SfJ "^Sf ^ an admission from him as. corroboration of the 
£r Srnnn ^' f"" '""u'^ bc judging the vcracity of the teacher's claim on the basis of 

S?S'£ sh"? h°«dS^°- ' ^^l''^ f .^'^ ^^^^ indicated reasonabfy 
unequivocally that she had been given a hard time by the principal. 

Sdwl^ • 7«ke further action which the major victim had already claimed 

s^aZt^ti^Sf f^ ?® ^"""^P^ ^ * Principles of Procedure whkh 

guarantee the nght of the evaluaaon team speak to anyone who might inform the shidv 
imphcitiy as inany times as we considered Necessary. We my KLTSerendy we mav 
have even made the evaluation more 'democratic'Zt it is myView tTat ouracSs w^e ^ 
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completely cpmpatible with the Principles of Procedure and the commitnaent to denaocratic 
evaluation practice, and certainly no less democratic than any alternative course of action* 
Perhaps we could have collected the principal's account of the conversation and taken the risk 
of making matters worse, but if the account was different, would we then be obliged to go back 
to the teacher for her response, back to the principal again ? 

This issue is partly a function of the attenuated nature of the case- Because the site had so few 
participants and the presence of the program relatively transient (a few weeks at a time) it was 
simply difficult to get testimony; there were few informants. The evaluation budget did not run 
to protracted observation either. 



Issue 2: Who v:as at risk? 

All parties (including the evaluators) are 'at risk' in an evaluation* But 'risk' is not a 
dchotomous variable* There are de^ees of risk which are very much^a fimction of the power 
(including access to information) which people have* It is apparent from the events described 
that the principal thought he had something to lose ftom the data. Though we should not 
simply take that at face value. It is possible that the prmcipal saw that the evaluation might 
threaten a program he yalued, failed to convince the teacher that her commitment to truth was 
foolhardy and fcaced her to falsify her testimony in order to protect her from her own folly - a 
bit p^nising, but not absolutely refutable on our evidence. The teacher's veto on further 
questioning of the prihcipal^made it impossible to check that mterpretation. On the evidence we 
have, which is the teacher's. testimony and our observations of her manner, it does not seem 
likely that the principal was acting on the teacher's behalf. 

The evaluation despite its aspirations to being 'democratic' probably shifted the balance of 
power towards the principal. That is, despite strict care m the observance of the Principles of 
Procedure the evaluation had already done what damage it might have done to the relationship 
between the teacher and the principal It was released interview data which had bmnght th^ 
teacher's point of view to the attention of the principal. Perhaps her situation was retrieved 
somewhat by the opportunity the principal took to change what she had said (in spite of the 
method he used)* But there was no.guarantee about that She was already on record as 
someone who was prepared to speak out critically, and that m itself had put the Language 
Project at that school in jeopardy, at least if we can believe her account of the threats which 
were made* Perhaps the program should have been terminated, but these were not appropriate 
reasons for its demise* 

The evaluation had incidentally threatened the future of the program, but more importantly 
appeaifcd to have shifted the balance of power in the school in favour of the principal. He now 
had several 'reasons' for terminating both the program and the teacher's employment, reasons 
which were incidental to die quality of the program and to the findmgs of the evaluation itself* 
As far as this part of evaluation itself was concerned, we were not in a position to find oiit any 
more about the program, or about the site effects we had set in train* But we had found out 
enough to feel uneasy about the alternative explanation of our effect on the principal-teacher 
relationship. We did not think that he would suddenly be enamoured of this outspoken and 
critical transient teacher* 

The gender relationsldp here is familiar according to the testimony we had the male principal 
domineering the untenured female staff member* Institutional power does reside with men, 
and women are subjected to shaky forms of employment Sexism is structural and ideological 
and is a reasonable general explanation of the jpo wer differentials which are important in this 
case* But I do not want to pursue tiie gender issue per se* Rather, I want to direct attention to 
intra- and extra-institutional power differentials themselves and thdr relationship v/ith tiie 
evaluation process* 
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issue 3: What are we doing here? 



observation, fa several evaluation p'oS Se be^n 



The report 



of substance) of Aose^ole S^Sa l^vSTL^^^'^"^ ^'^^^^^ ^« could not find any 
school system Aose yo^^^^^ the fringes of the 

could hanHl«» Wa f.d^.^A were otten more than others m and around schools 
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Limits of democratic evaluation 

The general difficulty we confinont here is what makes evaluation 'democratic*. In particular 
there is tiie problematic role of the •external* evaluator, the nature of his or her •insertion* into 
the political context of the case, and the way in which knowledge external to tfie ciise is brought 
to bear to inform it That evaluation is an ideological process which affects the distribution of 
power is obvious and widely documented. The best known encapsulation of tfie ideology of 
evaluation is MacDonaltfs (1976) division of evaluation into autocratic, bureaucratic and 
democratic forms. MacDonald is an advocate of democratic approaches to evaluation, tiiough 
his conception of what •democratic^ means in the context of •external^ evaluation 1 has been 
criticized by Lakomski: 

Just as democratic evaluation •takes for granted^ the political fiamewoik of 
liberal pluralism so it accepts uncritically the very program it evaluates. More 
specifically, it does not raise the question of how and why ihis program came to 
be conceived and implemented. Worse, it cannot even raise the question given 
its grounding in social phenomenology ... The consequence of the mability to 
account for the genesis of social and educational programs and to judge their 
worth leads democratic evaluation iniplicitiy to affirm the status quo. (Lakomski, 
1983, p. 273) 

There is an obvious overstatement here which I will deal with first Democratic (naturalistic 
case study) approaches to evaluation are not purely phenomenological for two reasons. First, 
the •case^ is often includes people who do attribute meaning to the case m the very terms 
Lakomski say they don^t And second, it is difficult to prise apart into separate realms of logic 
of the discourses of liberal pluralism and critical theory to which she is alluding at this point 
Concepts like hegen[K)hy, ideology, resistance can easily part of the ordinary discourses of 
some cases themselves. Nevertheless, the point Lakomski makes raises an important issue. 
TTiere are understandings and theoretical perspectives available which may go beyond those of 
the case participants and which might help to explain the •contained^ way in which the program 
IS understood The problem with purist phenomenological approaches to evaluation is that they 
do not conunit themselves to making these other potentially enlightening and educative 
perspectives an aspect of the reporting of the case. Their aspiration is to tell it like it is the terms 
and understandmgs of the most directly affected participants. The other side of that coin is the 
problep of •democratically^ making new knowledge available without at the same time 
practising the epistemological imperialism which sometimes characterises the woik of 
professional researchers. Both interpretivism and epistemological imperialism risk creating the 
same outcome: affirming the status quo as Lakomski has argued. 

In the instance and case cited above, the democratic approach probably did more harm than to 
confirm an inequitable status quo. There is a teal chance it made matters worse. 

The Lakomski analysis is obviously pitched at different level to the issue raised by the 
confrontation discussed above, though they are related m important ways. Lakomski was 
aware that even democratic evaluations are usually commissioned by policy makere in a social 
and political structure which surrenders to •bureaucratic rationality^ (Rizvi, 1986) much of the 
participatffry etWc and practice implicit m the ordinary conception of democracy. The question 
is whether evaluation practice might ever become sensitive to these local issues when it is so 
intmiately linked to tiie knowledge production apparatus of the institutionalised state. 

Lakomski suggested that the answer lies in recognizing that method of enquiry itself is value 
laden, and tiiat enquiry based on a cri tical epistemolngy is less likely to confirm and compound 
existmg differentials of po-.7er and status. She noted MacDonaltfs acknowledgement that 



^ Note that MacDonald (1978) has also argued for *self-evaluation* as an expression of the democratic impulse in 
evaluation 
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j"^^ ^^^o^^^^s within given programmes 
'^'l?'^®' programmes (surSmtive evaluation) 



She then went on to argue: 



v^ nS^Tnn/°i^ also examine those 

P"^^'*^ '^^eard to their ideological assumptions. 
But such critique of ideology cannot take place within the phenomenoloeical 

fn?S:f • 7^"' ^ '° ^ ^''^^^"^ ^ fa^°^ of a criticS epSdo^^d 
and theory of education. (Lakomski. 1983. p. 274. emphasis in originS) 

B^Te^''l Q7fi^i°i"o7°/ "Sf (for example, 

uemstem. 1976; Fay, 1975; Habermas. 1972. 1974. Cair and Kemmis 1986^ But thpr<' w an 

!S?SinthT^'H^'"T ^^^"^ "^ti^^ perspectives whicSU upon Se naS^ of 
Si^^^H- I'^P ^^^"i researcher and the researched. Most critical app^aches to 
educational enquiry (mcluding evaluation) recognize the importance of anjaS w^wer 

pS£t^^srre^»S4^^^^ 

Lv 5 ^^fT ^ historical-materialist amounts of why peopIeK&S the wm S 
they do? If the answer is to engage the practical political strigde for improvf mTnrS^^^^^^ 

*v -^fy 'l""'^^ ^'^^^'^ act of^search is tSSig^XaTru^e 
SStioi? " ' ^P^' '^'^ commitment to Kl^ge 

In Uie OTtical tiieory Uterature on rese. ach methodology which relates to education there are 
wide differences m the extent to which the researcher should fed SitS S S?with 
people, rather thmi merely study them.. The nature and stxtv^^oT^^^^t^S^^^^, 
ordy the nature of evaluation contracts researchers should beVreparS to S?ut 
researches will work at all m those 'external' evaluation roles. The diS of Sti^^^^ 
quite easily shown For example. Popkewitz builds the case for criticS^Sachw to 
aSsSa^r^^^^^ «ve app^«r takes an 

What I cannot argue for is the superiority of any single paradigm for considering 

the complexities of schooling. Each intellectual tratUtioS prov§es a paScdL ^ 

vantage point for considering social conduct. The different paradigiSscaSi enable 

Sf. ° and into the relationsWp ofAe^^Iemems to 

to tir«^hi;.i"nf '° ^°Pt ^ liberal-dem^ocratic p^^ctive 
to the problem of social science. (Popkewitz. 1984. p. 54) 

4^^5Ji"?J?hS^'„^ f '^'^''^ (*at is. research done by people (for 

mSiLn ^^^^ ^^/^'^ themselves) the assumption that reseSpSatois are 
miten^ rather than expressions of political ccDlsSlation leaves Popke^te S Sli? 

positively recommend is the development of a 'self-critical 
quality that enables researchers to scrutinize the assumptions, implications and 
consequences of their work'. What Popkewitz conspicuously^ to 
recoinmend IS how this 'self-critical quaUty' is to be achieved. Forme this 
rductence to embrace the fuU implications of its own central a^gu^ent'^ 
book's most obvious failing. Another is its failure to indicate ffidnd of 
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^ research methodology which would give concrete expression to the alternative 
view of educational inquiry it advocates and defends. (Carr, 1985, p. 232) 

This criticism by Carr is fundamentally a criticism of the role of the evaluator simply giving an 
account of people's work (even jofli democratic processes for the release and validation). It 
leaves knowledge production and reporting larjgely in the hands of the external evaluator. 

Other advocates of the critical approach have been much more explicit about the kind of 
evaluation and research commitments it entails. Comstock, for example, described it m these 
terms: 

Critical social research begins from the life problennis. of definite and particular 
social agents who may be individuals, groups or classes that are oppressed by 
and alienated from social processes they rminiain or create but do not control. 
Beginning from the practical problems of everyday existence it returns to that life 
with the aim of enlightening its subjects about umecogniswd social constraints 
and possible courws of action by wWch they may liberate themselves. Its aim is 
enlightened self-knowledge and effwtive political action. Its method is dialogue, 
and its effect is to heighten its subjects' self awareness of tfieir collective 
potential as the active ietgents of history... Critical research links depersonalized 
serial processes to its subjects' choices and actions wiA the goal of eliminating 
unrecognized and and contradictory consequences of collective action. 
(Comstock, 1982, pp. 378-379) 

Leaving aside the issue of the appropriateness of the language he uses, I want to explore how 
Conastock envisages the role of researchers (and evaluators) witih respect to program 
participants. Comstock went on to develop in somewhat programmatic form the way in which 
a critical researcher must woric (Df particular note was the commitrnent to 

Participate in a theoretically grounded prograrn of action which will change social 
conditions and, in addition, will engender new less alienated understanctogs and 
needs... Critical researchers do not, therefore, enter progressive groups on an 
episodic basis to solve cle^ly defined problenas. Since their aim is to stimulate a 
self-sustaining process of critical analysis and enli^tened action, it becomes 
necessary for critical researchers to ally themselves with progressive groups and 
work with them for considerable periods of time. (Comstock, 1982, p. 386) 

In tfiis view, the researcher comes closer to engaging the political struggle with the social actors 
m the situation under study* But as Kemmis (1983) has noted, Comstock still reserved a 
somewhat special and privileged place for the researcher who in important ways in this 
conceptualisation still controls the conceptualisation of plans* But several points Comstock 
made here are relevant to the question of what an (admittedly idealised) image of denxxaatic 
work by a researcher might be. First, the identification of the program in which the work to be 
engaged is not predicated upon the bureaucratic necessity of an levaluation'. Rather, research 
sldlls are put at the disposal of 'progressive groups' who have essentially defined themselves a 
priori. Second, the substance of the work is not determined by a bureaucratic definition of a 
case or the sphere of action supposedly influenced by the targeted dispersal of funds. Third, 
the duration of engagement is not determined by costing and review schedules but by the 
ordinary, progress of the work In turn these produce a quite different relationship between 
between the researcher and 'program' participants. In this view, we can no probably longer 
speak of 'programs' at all, and tenris like 'movement' might seem more appropriate. ^ 

If we accept the idea of dennxaracy that underpins the Cdmstock proposal, the state and its 
institutional infrastructure plays a less central role in the production of knowledge about 
programs of reform. What place is there for 'democratic evaluation' of the kinds practised by 



Note that the evaluation (conceived somewhat in traditional tenns) of PUSH/EXCEL was considered less than 
feasible by its J5)pointcd evidualbrs hecausfi it was a •movement* and not a program* 
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external' evaluators in forins of democratic life like the one Comstock proposes? Of course, it 
IS unfair to compare the harsh realities of one kind of practice with an unrealised ideal. 
Neverthless, we can recognise some important shortcomings of 'democratic evaluation' in a 
buieaucratised state. But how do we go about correcting them; via the fiirtlier refinement of 
democratic evaluation, or from a different tack altogether? 

A more recent proposal fi-om Lakomski (1985) asserted the importance of educational 
practitioners, playing a primary role in,educational enquiry. The critical epistemology which 
underpinned her advocacy was based on a rather dismissive critique of Habermas, but other 
authors, have used Habermas' theory of knowledge constitutive interests and critical theory 
more generally to develop an epistemology which seeks to justify action research (and self- 
evaluation) as an appropriate approach to program evaluation. In doing so these authors try to 
recast the role of the 'external' researcher. They also work from a different concept of 
democracy. 

Such approaches to evaluation argue that understanding the nature of the program (or 
m6veinent):is impossible if the struggle to change is liot directly engaged. That is, there is an 
explicit challenge to the 'externality' of the evaluator. In tliese approaches, the organization of 
political action aimed atemanciparion ftn use Habermas' Sfjmewh^t grand terml) is inextricably 
hnked with educational improvement and the growth of critical understanding. Eiiquiry is 
always an opporainitv. a commitment to address questions of dissdvantage, powerlessness, 
irrationality and injustice with action as well as through tlie research act itself. TTiese 
apittoaches have ari explicit politics directed not only at reducing the distortions.produced in the 
kinds of scenario jportrayed earlier, but at actively confi-onting with participants the power 
differeiitials which produce the distortions in the first place. The aim of such approaches to 
evaluation is not to correct the perceived failings of the relationship between evaluators, 
sponsors and the state by improving the ways knowledge production is controlled; but by 
chanring the way it is knowledge is 'produced' in the first place. But these approaches present 
us with another set of problems which may not mean that the outcome is any mote democratic 
than thiei conceptualisation it replaces.? 



The 'emancipatory critical' approach to democratic evaluation 

Kenarriis has attempted in several places (1980a, 1982, 1983, 1986, 1987) to develop the 
argument that there is a specific need to move towards emancipatory critical approaches to the 
evaluation of educational programs. In the emancipatory critical approach, the 'evaluator's' 
aspiration is not 'scientific objectivity' nor 'disciplined subjectivity' but to participate with 
others in a prcwess of collective action and self-reflection ~ the struggle to improve education 
through the aggregatiori of critically examined individual efforts conceived as part of a ^up 
commitment Partigipatipn in the evaluation process, in the critical communitv. informing and 
intormed by the evaluation, is crucial. It is argued that the change in focus is such that the 
l^guage of evaluation Is no longer appropriate. In more recent work, Kemmis and others have 
begun to move move the discourse away from the language of evaluation (except occasionally 
to refer to the notion of 'self-evaluation') and to argue instead for actionreseareh (Cafr and 
McTaggMt andGarbutcheon-Singh, 1986, 1988; Kemmis and McTaggart, 
1988a, 1988b). The idealisation.of action research envisages quite different notions of 
democracy from those implicit in the Principles of Procedure to which! have referred 

A particular kind of action research is envisaged - a kind which can actually help resist the 
kinds of domination sustained and confirmed by so-called 'democratic' approaches to 



It is also patronising and contradictory to think you are going to emancipate people more effecavely than they 
can do it for themselves. 

2 It is also possible that in practice democraUc evaluation (as the term is usually understood) as an aspiration is 
wiongheaded anyway. In the real politik of educational systems, people without power may be in stronger 
position after an 'evaluation' if they can discredit it by demonstrating that the^ views have been ignored, that 
they would be if their views have, ostensibly been taken into account 



" Vim 

eyaluatioD* In this kind of action research it is argued» researchers and evaluators might join 
otheri>7Steih participants in collective woik to improve education in quite different 
orgahisatibnal and social arrangement to those implied bv even the best practices of 'external' 
evaluation* In these forms of social organisation a key idea is die commitment to group 
eni fflgement in action and reflection* Otie key feature of tiiis wodc is the elevation of the status 
of the production of knowledge through the experience of educational and social woric 

Th^ is a somewhat problraiatic ideal underlying each of these concq)ts die ideal of 
'synunetrical conimunication' which derives &om Habermas conceptualisation of an 'ideal 
speech siiuation' (Brandt, 1978, 1979; Werner and Drechsler, .1978; Kemmis, 1980b, Carr and 
Kemmis, 1986). It means die use of discpurse to attempt 'to come to an agreement about die 
truth of a problematic statement or the correctness of a problematic norm carries with it a 
supposition that a genuine agreement is possible'. Habermas argued diat provided die 'rational' 
d^^ision is based solely on better arguihent, such agreement is possible. The characteristic 
demanded of discussion to achieve diis end is that it be 'free dom all constraints of dcmnation, 
whedier didr source be conscious strategic behaviour or communication barriers seeing in 
ideology or neurosis'. 'In particular, all participants must have the same chance to initiate, and 
perpetuate discourse, to.put forward, call into question, and give reasons for or against 
staterbents, explanations, mterptetations and justifications. Furthermore, diey must have die 
same chance to exprcfss attitudes, feelings, mtentions and the like, and to command, to oppose, 
to permit, to forbid, ete,' (McCarthy, 1978, cited in McTaggart and Fitzpatrick, 1981) 

The logical and pnctical possibilities of the 'ideal speech situation' where discourse is 'ficeed 
fix>m the constraints of action' in die interests of die pursuit of trudi have been challenged in 
several ways (Nielsen, 1983; Connolly, 1987). One cluster of criticism questions its 
fundamental consensualism, arguing iiistead diat all aspects of speech are fundamentally 
contestable, ^lodier related cluster of concern challenges the assumption that people will 
becon^ committted to discourses which, through their symmetry, act against their own interests 
(dirowing away die advantages diey have achieved because of die inequities capitalism creates). 
However, it is important to recognise diat die idea was presented by Hab^tnas as a working 
proposition (and now seems to attach less importance to it). 

We should not discard die possibility of more syimnetrical cominunicarions on the basis of such 
dieoretical analysis widiout at die same time examining die ways die idea has been taken up in 
practice, for example, dirough a general commitment to rationality duou^ 'reasonableness'. 
The possibilities for more symmetrical communications can appropriately be tested by 
participants in social and educational action dupugh reflection on the ways in which their own 
grou]^ processes facilitate or frustrate die efficacy of their action and die evolution of dieir 
understimdiiigs. We must recognise that concrete coercion, hegemony and curtailed 
understanding of many kinds get in die way, but die useful question is whedier diere is any 
evidence diat people can aspire to and begin, to recognize authentic improvement in practice, 
understanding and social organisation for diemselves. The examples.cf Paulo Rreire's wotk 
and ihe Nicaraguan Literacy Campaign (Lankshear, 1986) suggest diat it is reasonable to be 
optimistic oh that score. From that work we find another sense of the practice of democracy 
which is captured somewhat polemically by Giroux who uses the concept of die resisting 
intellectual' to identify those who want to work togedier to change existing patterns of social 
relationship and organisation and knowledge production: 

Resistiiig mtellectuals is a cate^ry diat draws from die insights of bodi Gramsci 
and Paulo Freire. This term is indebted to Gramsci's notion of die organic 
intellectual but differs in diat it sugg^ts that such intellectu^ can emerge from 
and work widi aiiy number of oppositional groups, odier dian and including die 
working class, d^at advance emancipatory traditions widiin and outside of 
alternative ipublic spheres. Central to the category of resisting intellecuials is die 
interplay of the languages of critique and possibility. Utiliang the language of 
self ditique, resisting intellectuals employ die dl^urse of self criticism as well 
as forms of critical analysis tiiat interrogate the Meological and material practices 
of domination. Furthermore, resisting intellectuals take dieir cue from Freire and 
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develop a critical vernacular that is attentive to the problems experienced at the 
level of everyday life, particularly as these are related to pedagogical experiences 
connected to classroom practice* The language of critique unites with the 
language of possibility when it points to the conditions for new forms of culture, 
alternative social practices, new modes of communication, and a practical vision 
for the future* (Giroux, 1986, p* 39) 

That is, according to tiiis line of argument, by being a member of an 'oppositional group* 
engaged in collective action and reflection, the 'individual' is in a position to both confront and 
unmask the power relations implicit, for example, m bureaucratic rationality, with some 
nieasure of safety - politically with respect to others, and psychologically with respect to 
himself or herself. The comparison with the exposure of the vulnerable mdividual (and 
vutaerable report) in democratic evaluation is significant 

Is there a role now for denaocratic approaches to evaluation governed by the principles of 
procedure mentioned earlier? Periiaps they ate a way of teachmg people the virtues of the 
explicitly self-reflective approaches* Efforts to move incrementally fix>m one paradigm to 
another are fraught with obvious danger* Sometimes the dangers are quite explicit For 
example, in the Victorian Education Ministry, the installation of corporate managerialism (under 
a Labour Party govemment) has tried to revamp condpletely the 'School Improvement Plan' 
(which had laboured long and hard to get action research gomg iii schools) so that it became 
responsible for its corporate vision of 'accountability'. One of the problems action research 
advocates must confront in situations like this is the cooption of all of the language and most of 
the important ideas of action research to subvert its fundamental aims* Nevdiheless, there is 
some evidence from several projects that a transition from democratic evaluation to critical 
community can be achieved in the context of 'assisted self-evaluation' (Keramis, 1986)* Within 
these projects there is a range of activities from conventional democratic evaluation approaches 
(for example, to help initiate self-reflection by identifying apparent contradictions in current 
practice), 'co-authorship' (Tripp, 1983), right though to explicitly emancipatory action 
research* It could be argued that because the diversity of activity is within each project a 
methodological paradigm shift has already occurred In that case the question still remains, but 
can be approached from a more general perspective* 

What are the events which can cause 'naturally occurring* groups to become self-critical? 
Individuals sometimes identify significant events which seen to act as 'triggers for 
transformation' (McTaggart and Garbutoheon-Smgh, 1986, 1988), but that view is 
individualistic and denies the importance of biographical trajectory and historical location* 
Retallick suggested that a group can become a critical community when participants are ready to 
examine their 

values as being interactively constructed, as constitutive of the community 
rather than being given, accepted or imposed* This suggests that a community 
might become critical* when participants regard their vsdues as objects of 
examination, interpretation, critique and reconstruction through dialogue* 
Values ai9 seen as problematic in a self-conscious sense* Perhaps all groups 
of people who share values to a reasonable degree at least have a potential to 
become critical communities, that is to say they are latent critical communities 
which can theoretically be transformed into self-conscious or critical 
communities* (Retallick, 1986, p. 4) 

In this view, the development of a critical conununity is not seen as a momentary transformation 
(as the tenn 'enli^tenment' may sometimes suggest), but ratiier the ongoing work of 
reconstructing a view of die world through action andreflection; Evaluation studies which seek 
to unveil (or at least postulate) die ideological underpinnings of the forms of organization, 
discourse, and practice in educational institutions may be quite helpfiil in creating the possibility 
for a group to beconie a critical conimunity, or for encouraging the actual formation of incipient 
groups. Of course, it should be noted tiiat such an evaluation approach has gone beyond 
phenomenology, and has taken a critical turn* But in the view of action researchers, interpretive 
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accpuhtSt even those involving ideology critique, are deficient They argize that critique of 
intciprctivism ajid critical hermeneutics must be sustained, the pioduction of reseaicn aid 
evaluation findings is not enough* As Mao Tse Tung once observed: 

If you want to know a certain thing or a certain class of things directly, you must 
personally participate m tiie struggle to change reality, to ch^ge tiiat tiring or 
class, of thmgs, for oiiljr tiius can you come into contact with Aem as 
phenomena; only through personal participation in tiie struggle to change reality 
can you uncover tiie essence of tiiat tiring or class of tirings and comprehend 
flienL (Mao Tse Tung, cited in Stenhouse, 1980, p* 43) 

What are tiie ways to achieve tiiat, by denaocratising evaluation practice or by refusing tiie 
potential coqptiveness of 'evaluation* roles in favour of a completely different conceptualisation 
of tiie relationship between researchers and researched? And if tiie latter, what are tiie 
discursive, practical, and organisational routes which lead in concrete practice away from 
he^mpnic and concrete constraints oh the one hand, and a disempoweringly polemic^ idealism 
on tiie otiier? 



The possibility of convergence « 

There are two perspectives taken here: one is a short exposition on tiie undemocratic distortion 
of instimtionalised form of 'democratic evaluation* apparentiy at least for undemocratic 
purposes; tiie other is a somewhat Ojptimistic set of propositions about tiie way in which tiie 
relationships rnight turn put if conceiyed in terms of a participatory conceptualisation of 
democracy. There is a risk of lapsmg mtp dualistic tiu'ridng here and of confusing strategic 
wisdond with cobption. The duaHsm can be approached by examining concrete efforts to 
draiocratise evaluation practiced Before proceeding I want to make anotiier point to complicate 
things* I have suggested above tiiat ^mocratic Principles of Procedure can bfe corrupted to 
distorttiieaccuracy of ail evaluation report. But otiicr things.can go wrong as well Fot 
exan^le, the criteria of fairness nright be used by participants who do not recognise aspects of 
tiieir behavioiff as racist to stifle an account of it (at least m tiiese terms). The criteria of 
relevance and accuracy raise sinrilar kinds of issues, which at the very least may profoundly tiie 
delay tiie evaluation report by repeated cycles of negotiation. And finally, whose criteria of 
fairness, relevance and accuracy are most appropriate and actually applied m tiie situation.? 

One of tiie difficulties we face is a real shortage of empuically grounded analyses of the roles 
democratic evaluation and action research have played in people's lives. From tiie perspective 
of democratic evaluation, Simons' Getting to Know Schools in a Democi-acv (1987) is tiie most 
thorough analysis of tiieory and practice, but it is historically containc/^ and I think as a result 
sees tiie possibilities for hroader social change as tilting at windrrdU^. there is also a chance 
that she is being realistic about the possibilities of access to prpgn* jis for evaluators. She 
quotes MacDonald and Nprris: 'It is difficult to change what you do not understand' (p. 83). 
In recent tirhes I have handed back a standard government contract for an evaluation basically 
because tiie government department negotiators would npt amend tfie contract to mclude 
Principles of Procedure like tiiose appended Though one negotiator considered tfie Principles 
'wholesome', tiie government department would not agree to allow unfettered publication once 
tiie report was considered fair, relevant and accurate. I often wonder late at night if I did the 
appropriate thing. Wasn't tiicre a chance even there tiiat I nright work in educative ways, and if 
necessary arpuhd tiie constraints of tiie contract? I still don't tiiink so because of tiie political 
climate at the tinie, but ... 

In acticMi research, I find tiie best documented account is still tiie Ford Teaching Project (Elliott 
and Adehnan, 1973a, 1973b), but Adelman (1989) hunself laments tiie lack of appropriately 
documented exarnples of action research. Especially lacking are accounts of projects which 
actually examine the claim that action researches have become 'emancipated' or 'critically 
conscious' as a result of tiiehr activities (see Grundy, 1987). We mi^t find better examples of 
tiiat in cross-cultural participatory (action) research literature (Lankshear^ 198Q, but it may be 
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that in those situations concrete oppression is more important than hegemony and is accordingly 
more accessible to understanding and amenable to resistance. 

Because of Jie way power is distributed in the hierarchical, bureaucratic systems typical in 
education, the systemic role even of 'democratic' cvaluators may always be to strencthen the 
hand of semor bureaucrats. As the instance cited in this paper suggests, there are Iwal effects 
as well. Even the most democratic evaluation may be less help to the already disadvantaged 
thaii system-level analyses of the relationship between knowledge and power (like Lako^'s) 
WLU show. Furthermore, democratic evaluators may not discovo- the reflexivity of their work, 
nie scenano outiined m this paper lurched into the perceptual field of the evaluators ahnost by 
Chance. In general, such issues are unlikely to surface in accessible forams, and even when 
they do, evaluators are hkely to have disappeared ftom the scene. 

^mocratic evaluators and action research advocates are generaUy agreed that an important goal 
°f *eir work is to help people to engage .the sttuggle to change themselves, their situations. 
«Jid their relationships with coUeagues, institutions and communities. They differ in that action 
research advocates m the cntical tradition see critical theory as a useful resource, and in a 
commitaent to engage more diiectiy in the poUtical struggle which attends and infomis the 
research act. It is a moot pomt as to which approach would immediately or ultimately make 
educators a^d education itself less vuberable to the hierarchical failings of die bureaucratic 
systems and mstihitions in^^which they work. My sense is that we ought to start as we mean to 
go on, by supporting people in their quests to improve themselves, but I think we must keep on 
nn^SlS "i^^^ improveineht should be judged,:anJ the limits on the kinds of 

opportumties afforded by different kinds of enquiiy. That, I think, is the critical project, and 
while demooatic eyalimtion might provide some opportunities. I leaUy think it will take a more 
oubstandal cl^ge in aiucational enquiry to achieve.it We need to think not only in terms of 
the vahdity of knowledge clauns but also of the vaUdity of research as adcmocraric act TTiat 
K^?! not only to change evaluation practice, but to work for change 

m the institutional and soaal contexts in which evaluation occurs. 
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APPENDIX 

PRINCIPLES OF PROCEDURE^ 

a. Indepe!«dence 

(i) No participant in the project will have privileged access to the data of the evaluation. 

(ii) No participant will have a unilateral right or power of veto ever the content of the report. 

b. Disinterest 

(i) The evaluators will attempt to represent, as widely as possible, tiie nmge of viewpoints 
encountered in the evaluation, ratiier tiian to enunciate their own perspectives or private 
views. 

(ii) The evalpators recognize that explicit or implicit recommendations appearing in reports 
wiU not be regarded as prescripHons by programme participants. As far as possible, 
however, the evaluators will atteinpt to present recommendations from participants ratiier 
tiian to use the evaluation as a platform for tiieir own preferences. 

c. Negotiated access 

The evaluation will seek only 'reasonable access' to relevant data sources. The evaluators 
will assume mey can freely approach any individual involved in tiie project to collect data. 
TTiose approached should feel free to discuss anv matters they see fit All such 
discussions will be treated as privileged by the evaluators. The evaluators are bound to 
portray the project and tiie issues it raises, but die release of specific information likely to 
Identify informants will be subject to negotiation witii these infonnants. 

d. Negotiation of boundaries 

(i) The evaluation will.be issues-oriented. The principles for inclusion of concerns, 
perspectives or information in tiie study or its reports are tiiat tiiese concerns, perspectives 
or Items of infomiation contribute to understanding tiie project, especially in so far as it is 
variously understood by participants in and observers of tiie project from tiieir different 
pomts of view. A major task for die evaluation, tiierefore, is to attempt to piece tiiese 
disparate perspwtives togetiier into a coherent (tiiough not necessarily syhtiietic or 
complete ) account of tiie project as a whole. Thus, according to tiiis principle of 
mclusion, tiie perspectives of all participants and interested observers have a 'right to be 
considered in tiie evaluation. 

(ii) The princ j)le for exclusion of concerns, perspectives or infonnation is tiiat tfiey can be 
shown to be false or unfounded, irrelevant to tiie project, or to unfairly disadvantage 
mdividueJs or groups involved witii die project. 

e. Negotiation of accounts 

(i) The criteria of fairness, relevance and accuracy fomi tiie basis for negotiation bemeen tiie 
evaluators and piarticip^ts in tiie study. Where accounts of tiie work of participants' 
mvolvement m tiie project can be shown to be unfair, irrelevant or inaccurate, tiie report 
wll be amended. Once draft rcpoits have been negotiated with participants on the basis 
of fliesc.cntena, tiiey will be regarded as having tiie endorsement of tiiose involved in Uie 
negotiations witii respect to fairness, relevant and accuracy. 



1 Adapted from Kenunis,S., and Robottom, I. (1986). 
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^ ^ ib^Wo 2 ®^ ^''^""^ ^» necessary, be phased to protect 

information flow Parts of a report may 
nf.o?.S5!^KT* ^i^T' ">.*>^duals who could be disadvantaged if the ^rt were 
negotiated as a whole with all participants. 

f. Negotiation of release 

(i) Thwe wiU be no secret reporting. Reports wiU be made avaUable first to those whose 

^r^? fP'^'S"' ^"^^"^PH ^ ^ members olSepSySnce 

will receive reports earliest, with other audiences receiving them later. """^^^^c 

rLS^P?^ ^'^P^ copulation beyond the community of interests formed by 
° .S® f"^ *® evaluators is a matter for negotiation and 

decision witiim tius community of mterests. Given that the reports hive be^n 'endorsed' 

accurate by the procedure of negotiation ofaccounts. rdeas^ rfa 
nSr?o^^°'^T°'" u^^' audiences would unfairly disadvantage any member of the 
pnnmy audience.^In this case, an amended version of the report may be prepared vSdch 

b^t^Z'^T ^ ^- ^ °f report wSKS: 

by Its releasc disadvantage any member of the prima^ audience. Any such amended 

account of the project, however. . uv uxo«> 

foregoing ppiciples of procedure, the circulation of reports wiU be 
restncted unless the report has been cleared for-unrestricted circulation. nSSctions on 
circulation will be clearly indicated on the cover pages of all reports. 

g. Publication 

H Circulation only in the fonn estabUshed by the 
procedure of negotiation of accounts; that is. they must be 'endorsed' by the members of 

proSSS* P"™"""* Producal according to these principles of 

fet*&X,~l"^'"'°" ^"^ of 

h. Confidentiality 

(i) The evaluators wiU not examine files, conespondence or other documentation vnthout 
explicit authraization and will not copy from those sources without permission 

Jit^nS °!f exchange.-? will not be considered 'off-the-record'. but 

those mvolved are ft-^. both before and after, to restria aspects or parts of such 
Sw^S^' ^ correct or improve their statement Quotations, veifcatim ttanscripts and 
S '^^^ or recommendations, where S arr 

* f y Identify their sources, will be used in reports only with the 
KoS? mfpnnaiit (i.e. the authorization achieved by the procedure of 

negotiation of accounts). Where ihfonnation is general or where the sources are 
be soight' identification pf specific individuals, no clearance will 



(ii) 
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Oii). the cyaluators arc responsible for the confidentiality of data collected by them in the 
course of the evaluation. 

Ov) In general, it should be noted that these confidentiality rules cannot be used to withdra sv 
reports fixma general view; once fair, relevant and accurate accounts have been released 
and when they are presented in ways which do not unnecessarily expose or embarrass 
participants, such reports should no longer be sheltered by flie prohibitions of 
confidentiality. 

i. Accountability 

The evaluators cannot make all their tea^ds publicly available without breaching the 
evaluation's principles of ps^cedwie. Nevetthebss, tiie evaluation and tiie evaluators 
must be accountable to sponsors, project pjuticipants and the evaluation audiences. Thus: 

(i) The evaluation will keep appropriate financial and administrative records which will be 
open to its immediate sponsor. 

(n) The evaluation will be accountable to participants as outiLied in these principle;; of 
procedure. 

j. Agreement to these principles of procedure 

<i) The evaluators Cfthfipt be held responsible for breaches of these principles by others 
ifivolvcd in the evaluation. !♦ is the responsibility of members of the primary and other 
audiences of the evaluation to respect the confidentiality of reports and any restrictions on 
theu: circulation. 

01) In commissifiniftg this evaluation study and in accepting the commission, the sponsor and 
the evaluftt&rs agree to abide by these principles of procedure. 
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